Benchmarks Website V3: Admin and Auto-Deploy#7849
Conversation
| @@ -0,0 +1,22 @@ | |||
| # SPDX-License-Identifier: Apache-2.0 | |||
| let bearer_token = | ||
| env::var("INGEST_BEARER_TOKEN").context("INGEST_BEARER_TOKEN env var must be set")?; | ||
| let admin_bearer_token = env::var("ADMIN_BEARER_TOKEN").ok(); | ||
| let bind_addr = env::var("VORTEX_BENCH_BIND").unwrap_or_else(|_| "127.0.0.1:3000".to_string()); |
There was a problem hiding this comment.
nit - make port configurable? can default to whatever but respecting PORT is usually useful
| snapshot_dir = %snapshot_dir.display(), | ||
| "bench server listening" | ||
| ); | ||
| axum::serve(listener, app).await?; |
There was a problem hiding this comment.
realized I commented on tests before - but having ctrl-c handling here is a good practice, gotta make sure it actually shuts down.
There was a problem hiding this comment.
I just use pkill directly on the server, and if the admin wants to have a server that they can SIGINT, then they might as well just do cargo run vortex-bench-server?
1477314 to
07656cc
Compare
Merging this PR will not alter performance
|
3bc4736 to
6ffb3dc
Compare
c26d766 to
03f8731
Compare
6ffb3dc to
a246b73
Compare
|
You'll need to delete https://github.com/vortex-data/vortex/blob/develop/.github/workflows/publish-benchmarks-website.yml, otherwise it will publish a new image and break the live website |
|
I will figure out the performance problems before I merge this |
c35640d to
9625ab4
Compare
a246b73 to
1270615
Compare
9625ab4 to
ee5f352
Compare
02a5b97 to
499fe8f
Compare
ee5f352 to
1c1dab6
Compare
84e51d8 to
bc08b17
Compare
c49d7c2 to
0e8a42d
Compare
Replaces the ad-hoc SSH-and-`nohup` deploy of the v3 benchmarks site
with a systemd timer that polls origin/develop every 60s, builds and
atomically swaps the binary, and verifies /health. Adds an hourly
gzipped-snapshot timer and two server-side admin endpoints so backups
and ad-hoc reads no longer need to stop the server.
Two new routes mounted only when ADMIN_BEARER_TOKEN is set:
- POST /api/admin/snapshot?ts=<id>: runs `EXPORT DATABASE … (FORMAT
csv)` against the live DuckDB connection, into a fresh subdirectory
under AppState::snapshot_dir. ts must match [A-Za-z0-9_-]{1,64}.
CSV is the only EXPORT format that ships with libduckdb-sys's
`bundled` feature; flipping to parquet or a Vortex layout later is
a one-line change.
- POST /api/admin/sql {sql, ?format=json|table}: runs read-only SQL
(SELECT/WITH/PRAGMA/SHOW/DESCRIBE/EXPLAIN, anything else 403) and
renders either JSON or a duckdb-cli-style ASCII table. Uses the
same connection mutex as ingest, so a slow SELECT briefly delays
writes.
Auth is independent of the ingest token (separate ADMIN_BEARER_TOKEN
env var) so the two rotate separately. Both use constant-time eq.
Everything an EC2 host needs lives under benchmarks-website/ops/:
- install.sh: idempotent one-time bootstrap (state dirs under
/var/lib/vortex-bench, sudoers fragment, env-file template, systemd
units, enable + start the timers). Recommended first-time path is
"wait for the deploy timer to build, then run migrate.sh"; preserving
an existing $HOME/bench.duckdb is documented as a side note.
- deploy.sh: called by vortex-bench-deploy.timer every 60s. Cheap fast
path (sha == stamp → exit 0). Path filter on benchmarks-website/{server,
migrate}, Cargo.toml, Cargo.lock — vortex-array PRs fast-forward the
working tree but skip the rebuild. Atomic versioned-binary symlink
swap, sudo systemctl restart, /health verification with rollback to
previous binary on failure, stamp updated only on success so failures
retry on next tick. Keeps last KEEP_BINARIES (default 3) versions.
- migrate.sh: stops server, snapshots current DB to bench.prev-<ts>.duckdb,
passes args through to `cargo run -p vortex-bench-migrate --`, restarts.
- backup.sh: hourly. Calls /api/admin/snapshot, `tar czf`s the CSV
directory into <ts>.tar.gz (gzip reclaims ~5–7× on this shape since
most data lands in BIGINT[] runtime arrays serialised as text), uploads
with `aws s3 cp` to s3://vortex-ci-benchmark-results/v3-backups/, and
cleans up both local copies. Logs the compression ratio so a future
regression shows up in `journalctl -u vortex-bench-backup`.
- inspect.sh: thin wrapper around /api/admin/sql, no server stop.
- systemd/ units: server (Type=simple, Restart=on-failure, hardening
via ProtectSystem=strict), deploy oneshot + 60s timer, backup
oneshot + hourly timer (Persistent=true so a missed hour catches up
after reboot).
A symlink at /var/lib/vortex-bench/ops -> .../benchmarks-website/ops
keeps the systemd ExecStart paths stable as the repo location changes.
- ops/README.md: full operator runbook — first-time install, day-to-day
ops (push to develop → live in 60s, monitor a deploy, force a deploy,
re-run migration, ad-hoc SQL, backup/restore, token rotation), and
failure modes (deploy retry loop, /health stuck, disk filling up,
backup not running, host migration). Targeted at a fresh admin who
has never seen the system before.
- benchmarks-website/README.md, benchmarks-website/AGENTS.md: updated
to point at ops/ and to reflect the systemd-based deploy.
- server/src/{lib.rs, app.rs, main.rs, admin.rs}: module map, route
table, env-var list, and admin module doc all updated.
The previous v3 docker artifacts are removed:
- benchmarks-website/ec2-init.txt: replaced by ops/README.md.
- benchmarks-website/server/Dockerfile: v3 isn't containerised any more.
- benchmarks-website/server/scripts/backup.sh: replaced by ops/backup.sh.
The v2 React/Vite stack is untouched. docker-compose.yml is left in
place; its v3 service entry is now orphaned but harmless and the v2
service is unaffected.
server/tests/admin.rs (9 tests):
- SQL round-trip (JSON + ASCII table format)
- Read-only allow-list (DELETE/UPDATE/DROP/INSERT/CREATE/ATTACH → 403)
- Allowed verbs (PRAGMA/SHOW/DESCRIBE/EXPLAIN/WITH)
- Bearer enforcement: missing/wrong/ingest-token-on-admin → 401
- Admin router not mounted when ADMIN_BEARER_TOKEN unset → 404
- Snapshot creates the export dir + schema.sql
- Snapshot of an existing dir → 409
- ts validation: empty / "../oops" / "with space" / 65 chars → 400
cargo test -p vortex-bench-server passes (admin: 9, rest: 18 pre-existing).
cargo clippy -p vortex-bench-server --all-targets --all-features clean.
Signed-off-by: Claude <noreply@anthropic.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Switches /api/admin/snapshot from `EXPORT DATABASE … (FORMAT csv)` to
per-table `COPY (SELECT * FROM <table>) TO … (FORMAT vortex)`. Dogfoods
the project's own format and compresses an order of magnitude better
than gzipped CSV on this shape (BIGINT[] runtime arrays + short strings).
- schema.rs grows a `pub const TABLES: &[&str]` so the snapshot loop
has a stable list to iterate.
- admin::snapshot writes `schema.sql` from `SCHEMA_DDL` verbatim, then
`INSTALL vortex FROM community; LOAD vortex;` (idempotent — autoload
is enabled in the bundled libduckdb-sys), then one COPY per table.
- ops/README.md restore section rewritten: untar → `.read schema.sql`
→ `INSERT … SELECT * FROM read_vortex(<file>)` per table.
- Two snapshot tests are marked `#[ignore]` because they need outbound
network to fetch the vortex extension. Run them by hand before merge:
cargo test -p vortex-bench-server --test admin -- --ignored
Signed-off-by: Claude <noreply@anthropic.com>
…docs
- build.rs runs `git rev-parse --short=12 HEAD` at compile time and
exports VORTEX_BENCH_BUILD_SHA, with rerun-if-changed on .git/HEAD
and .git/refs/heads so the SHA stays fresh across deploys. Falls
back to "unknown" outside a git checkout.
- HealthResponse grows a `build_sha` field (`&'static str`) and the
/health handler populates it from env!(). Lets an operator run
`curl /health | jq .build_sha` and compare against
`cat /var/lib/vortex-bench/last-deployed-sha` to confirm the live
process is the one the deploy timer last rolled out.
- ops/README.md grows two sections under day-to-day:
- "Which build is actually running?" — three identifiers, increasing
levels of certainty.
- "How do I manually rebuild and restart, outside the timer?" —
three knobs (restart-only, force-deploy, manual checkout-and-build
with the timer paused).
Signed-off-by: Claude <noreply@anthropic.com>
The deploy timer's fast-path exit when `origin/$DEPLOY_BRANCH` hasn't moved is the right default — but it means there's no way to say "I want a fresh build of whatever's on the branch I'm tracking right now, even though origin hasn't moved." Real situations that hit this: - Flipped DEPLOY_BRANCH and want the new tip within seconds, not on the next commit there. - Edited /etc/vortex-bench.env in a way that needs a rebuild (build flags, target paths) rather than just a `systemctl restart`. - target/ got wedged and you want a clean rebuild from scratch. deploy.sh now accepts either `FORCE=1` in the environment or a `.force-rebuild` sentinel file under `$STATE_DIR`. Either path skips the stamp comparison so the full build → atomic swap → restart → /health cycle runs against `origin/$DEPLOY_BRANCH`. The sentinel is consumed on consumption so the very next ordinary timer tick is a normal no-op again. ops/force-rebuild.sh is a one-liner wrapper that drops the sentinel and triggers the deploy service — operator-facing, no env vars or systemd-override invocations to remember. ops/README.md grows a "How do I manually restart or redeploy?" section with three knobs in increasing order of work: restart-only, deploy-if-moved, and force-rebuild. Drops the earlier "manual cargo build from a local checkout" path which was over-documented; the force-rebuild + DEPLOY_BRANCH flip covers the same ground without encouraging hand-edited build state on the host. Signed-off-by: Claude <noreply@anthropic.com>
`sudo systemctl restart vortex-bench-server` is silent on success, which is awful UX when you actually want to know the new process came up. Adds ops/restart.sh: snapshots the unit's MainPID + start time + symlink target + /health response, runs systemctl restart, waits up to 30s for the new process to answer /health, prints the same snapshot again, and exits 0/1 with a clear RESTART OK/FAILED line so the operator (or a script) doesn't have to guess. Sample output in the runbook section "How do I restart the server and *see* that it actually restarted?". Signed-off-by: Claude <noreply@anthropic.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com> (cherry picked from commit d5c6417a5686b20f62fbb315a6dd48f7e18f5174)
560e059 to
b133658
Compare
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Summary
This is the (hopefully) the last PR for the new benchmarks website, and you can see the prototype here: http://ec2-18-219-54-101.us-east-2.compute.amazonaws.com:3000/
With this PR, an admin can simply git clone the vortex repository on an EC2 instance, set the
INGEST_BEARER_TOKENandADMIN_BEARER_TOKENaccordingly (these are set as GitHub secrets in the repository), and then run just a few commandsHere are some of the features of the server itself:
vortex-bench-deploy.timerpollsorigin/$DEPLOY_BRANCH, rebuilds + atomically swaps the binary if website code changed, verifies/health, and rolls back on failure.vortex-bench-backup.timercalls a newPOST /api/admin/snapshotendpoint (CSVEXPORT DATABASE),tar czfs the result, uploads the backups3://vortex-benchmark-results-database/v3-backups/<ts>.tar.gz.POST /api/admin/sqlruns read-only queries on the backing database.Admin endpoints are gated by a separate
ADMIN_BEARER_TOKEN, mounted only when the env var is set.Quick start for a new admin
There are only a handful of steps. The full walkthrough is in
benchmarks-website/ops/README.md.On the AWS console (I already did this):
VortexBenchV3Backupsfrom the JSON in the runbook → IAM → Roles → createVortexBenchServerRole(EC2 trust) → attach to the instance.vortex-benchmark-results-database→ Management → add lifecycle rule expiringv3-backups/after 7 days.On the EC2 instance:
Admin Cheat Sheet
Details
Cheatsheet
Status — "what's the state right now?"
Manual triggers — "do something now"
Logs — "what happened?"
Pause / resume the autopilot — "stop fiddling for a minute"
Config changes — "edit settings"
Recover from S3 — "restore a backup"
One-time setup — "fresh box"
The 5 commands you'll actually use day-to-day:
restart.sh,force-rebuild.sh,inspect.sh,journalctl -fu …,curl /health | jq. Everything else is rare.